Volume 112, Number 1, January-February 2007 

Journal of Research of the National Institute of Standards and Technology 

[J. Res. Natl. Inst. Stand. Teclinol. 112, 39-51 (2007)] 

''Once is Enough " in Radiometric Calibrations 



Volume 112 



Number 1 



January-February 2007 



Gerald T. Fraser, Charles E. 
Gibson, Howard W. Yoon, and 
Albert C. Parr 

Optical Technology Division, 
National Institute of Standards 
and Technology 
Gaithersburg, MD 20899-8440 

gerald.fraser@nist.gov 
cliarles.gibson@nist.gov 
howard.yoon@nist.gov 
albert.parr@nist.gov 



The successful development of an Optical 
Technology Division quality system for 
optical radiation measurement services 
has provided the opportunity to reconsider 
the existing calibration procedures to 
improve quality and reduce costs. We 
have instituted procedures in our 
calibration programs to elimhiate 
uninformative repetitive measurements 
by concentrating our efforts on controlling 
and understanding the measurement 
process. The first program in our 
calibration services to undergo these 
revisions is described in this paper. 
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1. Introduction 

The successful development of an Optical 
Technology Division quality system for optical radia- 
tion measurement services has provided the opportuni- 
ty to reconsider the existing calibration procedures to 
improve quality and reduce costs. This effort leverages 
major advances in controlling manufacturing quality 
led by W. E. Deming, in Japan, following World War II 
[1,2]. These advances are equally apphcable to the 
delivery of services such as calibrations. Deming and 
others developed approaches for controlling industrial 
processes to reduce product variability and, conse- 
quently eliminate costly final product testing. For suc- 
cess, their approach requires manufacturers to under- 
stand and control all the steps in the production process 
so that variability in the final product is minimized to 
some desired tolerance level [3]. 

The early proponents of quality manufacturing 
developed statistical process control procedures and 
tests to help achieve low product failure rates. Their 
work has had a lasting impacting on efficiency and 
quality in product manufacturing. The control chart 



concept of W A. Shewhart used for tracking some 
measurable output of a process is particularly relevant 
for the present discussion [4]. Shewhart stressed the 
understanding of the whole process and control of the 
variability in the elements of the process, to minimize 
inefficient final product testing. Deming made this 
point, as well, in his renowned 14 points of manage- 
ment, reproduced in Table 1 . These points served as the 
foundation for the quality system that Deming devel- 
oped. Points 3 and 5, in particular, are relevant to the 
improvement of the calibration process as they target 
the elimination of piecewise inspection, emphasizing, 
instead, control and improvement of the entire process. 
Calibration services, like manufactured products, 
can be delivered with improved quality through effec- 
tive control of the entire process. Such control elimi- 
nates the needless inefficiencies endemic in calibra- 
tions performed by government and industry scientists, 
whereby the entire measurement is typically repeated 
three or more times and averaged to obtain a reported 
result. Calibration scientists justify the repetition 
by claiming that the statistical uncertainty in the 
final result is lower, confidence is improved, and it 
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Table 1. Deming's 14 points of quality management 



1. Create constancy of purpose to improve product and service. 

2. Adopt new philosophy for new economic age by management learning responsibilities and taking leadership for 
change. 

3. Cease dependence on inspection to achieve quality; eliminate the need for mass inspection by building quality into 
the product. 

4. End awarding business on price; instead minimise total cost and move towards single suppliers for items. 

5. Improve constantly and forever the system of production and service to improve quality and productivity 
and to decrease costs. 

6. Institute training on the job. 

7. Institute leadership; supervision should be to help do a better job; overhaul supervision of management and 
production workers. 

8. Drive out fear so that all may work effectively for the organisation. 

9. Break down barriers between departments; research, design, sales and production must work together to foresee 
problems in production and use. 

10. Eliminate slogans, exhortations and numerical targets for the workforce, such as 'zero defects* or new productivity 
levels. Such exhortations are diversory as the bulk of the problems belong to the system and are beyond the 
power of the workforce. 

11. Eliminate quotas or work standards, and management by objectives or numerical goals; substitute leadership. 

12. Remove barriers that rob people of their right to pride of workmanship; hourly workers, management and 
engineering; eliminate annual or merit ratings and management by objective. 

13. Institute a vigorous education and self-improvement programme. 

14. Put everyone in the company to work to accomplish the transformation. 



has always be done this way. Such repeat measurements 
are, in part, a legacy of metrology's early history when 
measurements where highly subjective, often involving a 
visual comparison. Details of this history for photo- 
metric and radiometric measurements will be given in 
Appendix 1. Here, we discuss the fallacy of maintaining 
such an approach and consider an alternative strategy 
whereby the calibration is performed only once. We 
denote this approach for convenience as "Once is 
Enough." 

To demonstrate the benefits of initiating a "Once 
is Enough" measurement strategy we consider one of the 
most popular optical radiation calibration services 
offered by the Optical Technology Division, the determi- 
nation of the spectral irradiance of an appropriate lamp 
standard. Historically, the spectral irradiance of the 
test lamp has been measured three or more times over the 



entire 250 nm to 2400 nm wavelength region. The varia- 
tion in the three measurements was used to assign a 
reproducibility uncertainty to the calibration, of unclear 
physical origin. 

Despite extensive automation, repeat measurements 
are time consuming and needlessly tie up expensive 
instrumentation. They also incur approximately three 
times the wear on reference standard lamps, customer 
lamps, and motion controllers such as monochromator 
translation stages and scanning grating drives. 
Ultimately, calibration service customers pay for the 
extra cost associated with these additional measure- 
ments, without receiving any significant benefit. These 
customers tend to follow NIST's lead and, likewise, 
perform repeat measurements with little benefit to "their" 
customers. 
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The repeating of a measurement is usually unneces- 
sary for a well-characterized process, with a clearly 
understood and verified uncertainty budget for artifacts 
and a well characterized and understood stability and 
repeatability It is only useful if it adds value to the state 
of knowledge of the measured quantity commensurate 
with the additional effort required. W. J. Youden, in his 
classic book [5], "Experimentation and Measurement," 
comments on the repeating of measurements: 

* 'Many people seem to feel that there is some 
magic in the repetition of measurements and 
that if a measurement is repeated frequently 
enough that the final result will approach a 
''true" value. This is what scientists mean by 
accuracy. 

Suppose that you are in a science class and 
that the next two students to come into the 
room were a girl five feet ten inches tall and a 
boy five feet nine inches tall Let each student 
in the class of 30 students already in the class 
measure the new arrivals to the nearest 
foot. The answer for both is six feet. Has the 
repeated measurements improved the 
accuracy? " 

In Youden's example, only one measurement is 
required to answer the problem poised. Repeat meas- 
urements are only necessary if they improve our knowl- 
edge of the measure and commensurate with the addi- 
tional time and effort required. Over the last two 
decades, the spectral irradiance measurement service 
has improved significantly in performance, aided by 
advances in instrumentation, environmental control, 
and automation; by implementation of a quality system, 
and greater understanding of the intricacies of the 
measurement. The consequence of the improved per- 
formance is that the "reproducibility" component of the 
measurement is now small, eliminating the justification 
for repeat lamp measurements and providing the oppor- 
tunity to implement "Once is Enough." Below we dis- 
cuss more fully the implementation of the "Once is 
Enough" strategy for spectral irradiance. 

2. The Spectral Irradiance Calibration 
Process 



PEL is the lamp-type designation (not an acronym) of 
the American National Standards Institute (ANSI). A 
photograph of an PEL lamp is shown in Pig. 1 and a 
typical spectral irradiance curve, as measured 50 cm 
from a lamp, is shown in Pig. 2. The lamps are pur- 
chased from commercial sources and evaluated for suit- 
ability as standards [6]. The evaluation includes visual 
inspection of the bulb and filament structure, assess- 
ment of the spatial uniformity of the output, testing for 
the presence of strong atomic emission lines, and 
examination of the stability and relighting reproducibil- 
ity. The lamps are also seasoned by operating at the 
prescribed current for 10 h to 15 h prior to cleaning and 
storing for future calibration. 

The technical details of the calibration procedure and 
the associated measurement uncertainties are discussed 
in NIST Special Pubhcation SP250-20 [6]. The calibra- 
tion procedure and measurement uncertainties differ, 
somewhat, from that published initially some 20 years 
ago, and will be described in detail in an updated 
Special Publication [7]. 




The Spectral Irradiance Calibration Services 39030C 
to 39046C disseminate lamps with spectral irradiances 
measured as a function of wavelength. Typically, a 
1000 W quartz-halogen PEL lamp is calibrated, where 



Fig. 1. Photograph of a 1000 W quartz-halogen FEL lamp dissemi- 
nated as a standard of spectral irradiance by NIST measurement 
services 39030C to 39046C. 
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Fig, 2, A typical spectral irradiance curve for an FEL lamp such as pictured in Fig. 1 . The line is drawn to aid the reader. The 
spectral irradiance (optical power per unit surface area per unit spectral bandwidth) is plotted as a function of wavelength as measured 
50 cm from the lamp. 



Briefly, the spectral irradiance calibration compares 
the spectral output of a known lamp — the primary work- 
ing standard (PWS) — ^to that of an unknown lamp, the 
lamp under test (LUT). Measurements are performed in 
the recently completed, second-generation Facility for 
Automated Spectroradiometric Calibrations, denoted as 
PASCAL 2, shown schematically in Fig. 3. The facility 
has four lamp stages, one for the PWS and three for the 
LUTs to be calibrated. The PASCAL 2 instrument con- 
sists additionally of a spectroradiometer, integrating 
sphere, and filter radiometer all mounted on a moveable 
carriage to align the entrance of the integrating sphere 
with either the PWS or with one of the three LUTs. The 
spectral irradiance of the PWS is calibrated by compari- 
son with a blackbody, HTBB, whose radiance tempera- 
ture is tied, by a set of filter radiometers, to the absolute 
cryogenic radiometer through the Spectral Comparator 
Facility (SCF) [8] or the Spectral Irradiance and 
Radiance responsivity Calibration with Uniform Sources 
(SIRCUS) Facility [9], as described by Yoon, Gibson 
and Barnes [10]. 



Delivery of the calibration service involves a number 
of steps, designated 1 through 9, as summarized in 
Fig. 4. The process starts with the acceptance by NIST of 
a request for calibration, and ends with the sending of the 
lamp and report to the customer, and addressing any 
follow-up issues, such as customer feedback, or specific 
lessons learned that can be applied towards improving 
the overall process. Such an analysis helps ensure 
continual improvement of the service and is a critical 
component of the associated quality system. 

The core of the measurement process consists of steps 
3, 4, 5, and supporting step 9, used to maintain the stan- 
dards, with respect to the International System of Units 
(SI)[1 1]. These four steps consist of the physical measure- 
ments and the assignments of their measurement uncer- 
tainties. The development of a NIST calibration service 
entails the determination and validation of a complete 
uncertainty budget for the measurements. Validation is 
aided by international comparisons with other national 
metrology institutes. The uncertainty budgets are avail- 
able to customers and other interested parties, as part of 
the public documentation for the service [6, 8]. 
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Fig. 3. Schematic diagram of the second generation Facility for Automated Spectroradiometric 
Calibrations (FASCAL 2) with the scale realized from the high-temperature blackbody (HTBB) 
as collected by the integrating sphere receiver (ISR). For the ultra-violet wavelength regions, the 
photo-multiplier tube (PMT) is used instead of the Si or the InGaAs detectors. 
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Fig. 4. The nine steps required for the Spectral Irradiance Measurement Services. Steps 1 
through 8 are routinely performed for each calibration. Step now is periodically performed to 
evaluate the irradiance scales maintained by the PWS lamps. 
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3, Spectral Irradiance Calibration 
Uncertainty Budget 

The present uncertainty budget for the spectral irra- 
diance calibration service is summarized in Table 2. It 
varies slightly from that shown in Reference [10], but 
the differences are small and represent refinements in 
the determinations of specific components, rather than 
a substantial change in the overall measurement 
approach. We will briefly discuss these components to 
provide the context for the implementation of "Once is 
Enough." 

Lines 1 through 4 in Table 2 give the components 
contributing to the uncertainty in the radiance tempera- 
ture of the High Temperature Black Body (HTBB) 
during calibration of the PWS lamps. The radiance tem- 
perature of the HTBB is determined using a set of filter 
radiometers with responsivities tied to the cryogenic 
radiometer. The HTBB is operated near 2950 K, to 
approximately match the spectral irradiance of the FEL 
lamp, thus reducing errors from spectral stray light and 
detector nonlinearity The HTBB is well represented by 
a single radiance temperature, independent of wave- 
length, as determined by comparison against the small- 
aperture, NIST Variable-Temperature Blackbody 
(VTBB) [12]. The uncertainty associated with this con- 
clusion is given in line 2 of the table, as "HTBB spec- 
tral emissivity" When using the HTBB to calibrate the 
PWS, additional uncertainty components arise from 
spatial variation of the radiance temperature over the 



blackbody exit aperture (line 3), and from drift in the 
radiance temperature of the HTBB during the measure- 
ments (line 4). 

Line 5, "geometric factors in irradiance transfer," 
gives the uncertainty in the geometric factors required 
to define an irradiance level from the HTBB at the 
entrance aperture to the integrating sphere receiver of 
the spectroradiometer. The irradiance is given by 



■■L{Xj,^^^,,)KrlJD\ 



(1) 



where L{X, T,^^^^^ is the Planck radiance, T^^diance is the 
radiance temperature of the HTBB, Kr^^ is the area of 
the aperture in front of the blackbody, the modified dis- 
tance factor is D^ = d^-\- P -\- rgg^, where Tir^ is the area 
of the entrance aperture to the integrating sphere 
receiver, and d is the separation between the two aper- 
tures. The uncertainty in D is dominated by the uncer- 
tainty in the separation between the two apertures, 
since d»r and d » r^^. 

Lines 6 through 9 are the uncertainty components for 
the transfer of the blackbody-based irradiance scale to 
a PWS lamp by 



^A,PWS ~ JX ^ ^A.HTBB ? 

'-*A,HTBB 



(2) 



where £'a,pws is the irradiance of the PWS, 5';l,pws ^^^ 
Sx, HTBB are the spectroradiometer signals when measur- 
ing the PWS and HTBB, respectively, and fx is the 



Table 2. Uncertainty Budget for Spectral Irradiance Calibrations Using PASCAL 2 



Source of Uncertainty 



Relative Expanded Uncertainties (k = 2) [%] 



250 350 450 555 655 



900 



1600 2000 2300 2400 



1. HTBB temperature uncertainty (0.86 K at 2950 K (B) 

2. HTBB spectral emissivity (B) 

3. HTBB spatial uniformity (B) 

4. HTBB temporal stability (0.1 K / h) (B) 

5. Geometric factors in irradiance transfer (B) 

6. Spectroradiometer responsivity stability (B) 

7. Wavelength accuracy (0.1 nm) (B) 

8. Lamp/spectroradiometer transfer (B) 

9. Lamp current stability (B) 

Total uncertainty of the primary working standards 

10. Lamp-to-lamp transfer (A) 

11. Long-term stability of primary working standards (B) 
Overall uncertainty of the test with respect to SI units 



0.57 


0.41 


0.32 


0.26 


0.22 


0.16 


0.09 


0.08 


0.07 


0.07 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.07 


0.05 


0.04 


0.03 


0.03 


0.02 


0.01 


0.01 


0.01 


0.01 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.60 


0.60 


0.30 


0.30 


0.30 


0.30 


0.30 


0.30 


0.30 


1.00 


0.58 


0.26 


0.13 


0.07 


0.04 


0.01 


0.01 


0.01 


0.01 


0.01 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.10 


0.07 


0.05 


0.04 


0.03 


0.03 


0.02 


0.02 


0.01 


0.01 


0.01 


1.03 


0.80 


0.50 


0.45 


0.43 


0.40 


0.37 


0.37 


0.37 


1.02 


0.50 


0.30 


0.20 


0.20 


0.20 


0.20 


0.20 


0.30 


0.30 


0.40 


1.31 


0.94 


0.73 


0.59 


0.50 


0.36 


0.20 


0.16 


0.14 


0.14 


1.74 


1.27 


0.91 


0.77 


0.69 


0.57 


0.47 


0.50 


0.49 


1.11 



Note: The Type A or Type B evaluation of the uncertainty is indicated in parentheses. 
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detector linearity correction factor associated with the 
different magnitudes of the signals measured when 
viewing the PWS and HTBB. Line 6, spectroradio- 
meter responsivity stability, is the uncertainty compo- 
nent associated with the drift in the integrating-sphere/ 
monochromator/detector system between measurement 
of the HTBB (5'x,htbb) and the PWS (^x,pws)- It is 
estimated using the standard deviation of the irradiance 
responsivities determined for three PWS lamps. Line 7 
encompasses the effect of wavelength error in the 
monochromator drive on the spectral irradiance 
assigned to the PWS. The magnitude of this uncertain- 
ty component is strongly wavelength dependent, due to 
the shape of the spectral irradiance distribution of an 
PEL lamp. Line 8 contains contributions from uncer- 
tainties infi, stray light and spectral scattering. Line 9 
is the uncertainty due to current instability in the lamp 
power supply. Note that line 8 and 9 terms are small 
and do not significantly affect the results. The terms in 
lines 1 through 9 are all type B uncertainties, uncorre- 
cted with each other and with magnitudes assumed 
independent of the LUT being calibrated [13]. Since the 
PWS are measured against the HTBB only on a period- 
ic basis, minimizing efforts in this aspect of the calibra- 
tion process will only have minimal effects upon the 
total time spent in the calibration endeavor. The total 
uncertainty on the PWS irradiance calibration given in 
the table is derived from the root-mean-square sum of 
the individual uncertainty components. 

The irradiance of the LUT, £'a,lut? is determined by 
comparison against the PWS using 






[3] 



where Sx^lut is the spectroradiometer signal when 
viewing the LUT. Note that Sx p^g in Eq. (2) is not iden- 
tical to the Sx^ms ^^ Eq. (3), since the measurements 
were performed at different times. Lines 10 and 11 are 
the uncertainty components associated with this irradi- 
ance transfer from PWS to LUT, which occur at the cal- 
ibration of each LUT. Line 10 is the subject 
of this paper and is discussed in detail in the next 
section. Line 11 represents our experience on the 
stability of the irradiance scale of the primary working 
standard, as determined by repeat calibrations per- 
formed over several years, and intercomparison of 
PWS lamps [14]. 

Further work on the uncertainty determination will 
be carried out to formally take into account the correla- 
tions in the data. Correlations will come about because 
the individual wavelength determinations are correla- 



ted through the commonality of the reliance upon a 
common blackbody temperature. When these effects 
are frilly considered the uncertainties could possibly 
lessen. 

4. Lamp-to-Lamp Transfer 

Initially, lamps disseminated to customers from the 
PASCAL facility were calibrated in groups of 12, by 
comparing their irradiances to that of four PWS lamps 
over the desired wavelength interval, typically 250 nm 
to 2400 nm [6]. Three LUTs and one PWS was mount- 
ed in the four PASCAL lamp stations for each of the 1 6 
calibration runs required. According to the previous 
procedures, the LUTs and PWSs were permutated so 
that each LUT was calibrated four times, each time 
mounted in a different station and compared against a 
different PWS, also mounted in a different station. At 
each wavelength of the spectroradiometer the four 
lamps were measured and the wavelength was incre- 
mented. This process put about 6 h running time on 
each LUT. 

The approach was intended to eliminate the possibil- 
ity of disseminating a lamp calibrated against a PWS 
that had drifted out of calibration. It also attempted to 
reduce the effects of variations in the PWSs and in the 
lamp stations on the reported LUT irradiance values. At 
the same time it provided a set of four measured values 
for calculating a standard deviation for the repeatabili- 
ty of the calibration; a type A uncertainty component 
provided with the other uncertainty components in the 
customer calibration report. 

The process was later modified in response to cus- 
tomer demands for faster and less expensive calibrations. 
Each LUT was now calibrated three times against three 
PWSs, instead of four times against four PWSs. As 
before, the LUT station, PWS, and PWS station were 
varied for each calibration of a LUT. Improvements in 
lamp temporal stability, through better lamp manufactur- 
ing and selection, reduced the magnitude of the PWS 
lamp drift during the measurements, providing addition- 
al opportunities to reduce the calibration time. This 
drift, which approximately and linearly depended on the 
number of hours the lamp had been operated since cali- 
bration, was effectively eliminated by changing the 
calibration procedure so the spectroradiometer scanned 
through an entire wavelength range once for each lamp, 
set by the responsivity range of the detector being used. 
Significant time was saved over the old process of 
viewing each lamp prior to changing the wavelength. 
The modified process continued with the launch of 
PASCAL 2. 
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The requirement to have each LUT continue to be cal- 
ibrated three times was driven by the desire to derive a 
standard deviation from the repeat measurements, to 
quantify PWS and station variability The repeat 
calibrations also provided confidence in the calibration 
stability of the PWSs and equivalency of the lamp 
stations. Both of these rationales are insufficient for jus- 
tifying the extensive amount of time, effort, and instru- 
ment and lamp hours expended on repeat calibrations, 
particularly given the level of agreement between differ- 
ent calibrations of the same lamp. As shown in line 10 of 
Table 2, the wavelength dependent lamp-to-lamp trans- 
fer uncertainty of 0.2 % to 0.5 % {k = 2) is small, relative 
to the total calibration uncertainty, which ranges from 
0.49 % to 1 .74 %{k = 2). This uncertainty component is 
based on a large number of lamp calibrations performed 
on PASCAL 2. Since there is a large sample set based on 
similar lamps to determine the repeatability uncertainty, 
two additional measurements do not add, in any signifi- 
cant way, to the state of knowledge about the calibration 
uncertainty. A typical example of lamp repeatability is 
shown in Fig. 5, for an LUT alternately positioned in 
three different stations (2, 3, and 4) and calibrated 
against the same PWS positioned in station L The vari- 
ation in the lamp-to-lamp calibration is well within the 
lamp-to-lamp transfer uncertainty listed in Table 2. Also, 
the lamp-to-lamp repeatability is well inside the overall 
uncertainty of the measurement given in the last line of 
Table 2, and shown in Fig. 5. The low uncertainty on the 
lamp-to-lamp transfer provides the primary technical 
justification for implementing "Once is Enough." 
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Fig. 5. A plot showing the variation in the spectral irradiance 
assigned to an LUT positioned in stations 2, 3, and 4 of PASCAL 2. 
A single PWS mounted in station 1 was used for the measurements. 
Also shown in the figure are the total expanded (k = 2) uncertainties 
(sohd lines), along with the expanded uncertainties for the lamp-to- 
lamp transfer (dashed lines) from Hne 10 in Table 2. 



5. Implementation of "Once is Enougli" 

Here, we provide a strategy for successful imple- 
mentation of "Once is Enough." The four components 
described in detail below will ensure confidence by 
both calibration scientist and calibration service 
customers in the spectral irradiance values of lamps 
measured only once. 

Automation, Automation improves the repeatability 
of a measurement by eliminating subjective human 
factors associated with reading analog meters, aligning 
the spectroradiometer optical axis to the lamp position, 
setting the monochromator wavelength, fixing lamp 
current levels, and reading spectroradiometer output 
signals. As mentioned previously and discussed in 
Appendix 1, the large number of subjective readings 
and manipulations originally performed in radiometric 
measurements, and their associated potential for human 
error and ambiguity, played a major role in institution- 
alizing repeat measurements in the calibration services. 

The present PASCAL 2 system is completely auto- 
mated, with the only human intervention being the 
mounting and dismounting of the PWS and LUTs in 
their stations. The NIST-issued PEL lamps are mount- 
ed on brass, bi-post bases to be fixed in kinematic 
mounts, so lamp alignments are reproduced within the 
uncertainties without any adjustments. The axial posi- 
tioning and distance setting of the lamp stations have 
been previously established and are not routinely 
adjusted with a change in LUT or PWS. Since a non- 
imaging integrating sphere is used as the collection 
source, the alignment is relatively insensitive to the off- 
axis tilt of the integrating sphere aperture, with respect 
to the LUT. After the lamps are mounted, the currents 
are turned up to their set points with computer control. 
After allowing time for the lamp outputs to stabilize, 
the spectroradiometer is automatically positioned to 
each station and swept through the selected wavelength 
range. The spectroradiometer output signals are record- 
ed by the computer, converted to absolute spectral irra- 
diances, and inserted into a text file for insertion into 
the calibration report template. 

Uncertainty budget A rigorous uncertainty budget is 
critical for the implementation of "Once is Enough." 
Such an uncertainty budget should be validated through 
independent measurements that are ideally based on 
both similar and different measurement methods 
performed by independent scientists. Uncertainty 
budgets in radiometry and photometry have conven- 
tionally included a "repeatability" component based on 
a simple statistical analysis of repeat measurements. As 
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mentioned earlier in this article, this was the motivation 
for repeating the measurement on a particular lamp, up 
to three or four times, in the belief that the uncertainty 
could be decreased. This is somewhat misguided as 
Table 2, line 10, contains an estimate based on many 
measurements, of the sort, of uncertainty that pertain to 
the expected variation in lamp performance; and a few 
additional measurements do not improve the statistical 
quality at all. 

The exact origin of the repeatability uncertainty 
component is often not clear, but certainly includes 
contributions attributed to other components in the 
overall uncertainty budget. The spectral irradiance 
"repeatability" component, denoted as lamp-to-lamp 
transfer uncertainty in Table 2, includes contributions 
from lamp-current drift and spectroradiometer respon- 
sivity stability, which were already accounted for in the 
previous lines in Table 2. The "long-term stability of 
the LUT" uncertainty component is not considered in 
Table 2, as this is the customers responsibility to esti- 
mate when they use the lamp. We note that such double 
counting unnecessarily increases the magnitude of the 
reported uncertainties on the calibrated irradiances. 

To avoid double counting, a rigorous uncertainty 
analysis should be physics based, clearly tying each 
uncertainty component to the measurement equation. A 
physics-based uncertainty analysis provides a frame- 
work for developing a strategy to reduce the overall 
measurement uncertainty, and in particular, the compo- 
nents limiting measurement reproducibility. Without a 
true physics-based model, it is difficult to determine 
with a finite measurement set, whether the repeat meas- 
urements should be distributed normally about the true 
mean. 

As discussed above, for the PASCAL 2 measure- 
ments, the "repeatability" lamp-to-lamp transfer uncer- 
tainty component is small, relative to the total uncer- 
tainty. This repeatability component conservatively 
reflects the repeatability of the measurements as shown 
in Fig. 5. Its small magnitude and likely over estimate — 
due to double counting — indicates that little is con- 
tributed to the knowledge of the spectral irradiance 
of the LUT by repeating the measurement three times; 
certainly not commensurating with the additional time 
or effort expended. 

Measurement process controls. Carefully selected 
measurement process controls and associated control 
charts help ensure correct instrument operation, scale 
stability, and fmal calibration accuracy. They further 
help the calibration scientist immediately identify 
problems arising in the measurement, eliminating the 
expenditure of time and resources to complete a cali- 



bration that will fall outside of acceptable quality 
levels. Process controls and control charts also elimi- 
nate unnecessary final product testing, typically per- 
formed through a repeat calibration, to ensure measure- 
ment quality. As discussed in the Introduction, Deming 
and Shewhart promulgated the elimination of final 
product testing and the integration of statistical process 
control in the manufacturing process, to ensure final 
product quality and performance. 

Por spectral irradiance measurements, two levels of 
process controls are being implemented. These process 
controls provide either a gross or fine level of assessment 
of instrument, PWS, and LUT performance throughout 
the calibration process. The process controls directly 
associated with the measurement of the spectral irradi- 
ances are being documented in a series of three control 
chart sets to help provide early warning of potential 
problems in the calibration process. 

The first level of control occurs in the lamp selection 
process. A number of measurements are presently 
performed on a lamp prior to its acceptance into the 
calibration process. The spectral output of the lamp is 
examined to determine the presence and strength of 
atomic emission lines. These lines are noted in the 
calibration report. The lamp output is examined at 
654.6 nm to ensure that it varies by less than 1 %, for a 
± 1 ° angular displacement from the defined lamp axis, 
and that it drifts by less than 0.5 % over a 24 h period. 
The former measurement is performed to aid lamp 
users interested in realizing a spectral radiance scale by 
illumination of a highly reflective diffuser. 

Three check standard (CS) lamps, calibrated against 
the HTBB, are used to assess the stability of the PWS 
lamps. The CSs are used infrequently to ensure the con- 
stancy of their spectral irradiance values. The PWSs, of 
which there are a total of six, are periodically used to 
assign a spectral irradiance scale to the CSs. The spec- 
tral irradiances determined for the CSs are required to 
stay within one PASCAL 2 standard uncertainty (i.e., 
half the k=2 value from the last line of Table 2) of their 
initially calibrated value. The measurements are 
tracked in a set of control charts for easy referral; 
denoted here as Control Chart Set 1 . 

A second set of control charts. Control Chart Set 2, is 
being developed to track the spectral irradiance respon- 
sivity of the spectroradiometer as a function of time, 
defined as the ratio of the output voltage of the spectro- 
radiometer to the input spectral irradiance from a PWS. 
Such control charts only provide evidence of gross 
change in the performance of the radiometer since, frac- 
tional changes in the responsivity over time are larger 
than the relative standard deviation of the reported 
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spectral irradiances. An example of such a plot for two 
wavelengths is shown in Fig. 6, for PASCAL. The 
figure shows a nearly linear drift in responsivity with 
time, except for a sharp break, due to a change in the 
operating voltage applied to the photomultiplier tube. 
The drift is significant and is caused by a continual 
change in the photomultiplier tube sensitivity. Present 
silicon detectors used in PASCAL 2, between 350 nm 
and 1050 nm, have much less drift so that the change in 
spectral irradiance responsivity of the spectroradio- 
meter within this wavelength interval is dominated by 
the change in the absolute reflectance of the integrating 
sphere coating; on the order of 0.5 %/day for a newly 
coated sphere in the ultraviolet wavelength region to 
< 0.1 %/month for the visible and infrared wavelength 
region. Control charts will encompass one wavelength 
each in the ultraviolet, visible, and infrared to track the 
spectroradiometer performance, when used with the 
photomuliplier tube, silicon, and extended InGaAs 



detectors. The overall stability of the radiometer system 
is well understood and has a long recorded history that 
serves as an overall check on the system as significant 
deviations from the expected output of the system will 
be immediately recognized. 

A third set of control charts (Control Chart Set 3) 
will be developed to track the ratio of the spectral 
irradiance of the PWSs as a function of time, again 
at the same three wavelengths. In the absence of a 
stable monitoring detector to assess the absolute output 
of a PWS, "Once is Enough" calibrations will be 
performed by comparing 2 LUTs against 2 PWSs 
mounted simultaneously in the four PASCAL 2 lamp 
stations. Sequential calibrations will position the 
PWSs, so all four stations are tested in the two cali- 
bration runs. The relative irradiances between PWSs 
should be predictable from the uncertainties in Table 2 
if the instrument and the standards are performing 
as expected. 
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Fig, 6, Plot showing the change in the spectral irradiance responsivity of PASCAL as a function of time at 330 nm and 555 nm. The 
nearly linear growth in responsivity with time is due to change in the sensitivity of the PMT used at that time. For PASCAL 2, a more 
stable silicon detector is used for wavelengths between 350 nm to 1050 nm, so that the drift in the spectral irradiance responsivity is 
dominated by drift in the reflectivity of the integrating sphere coating. 
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The assigned spectral irradiances of the LUT and the 
PWS can also be checked by filter radiometers calibrat- 
ed for spectral irradiance responsivity Since filter 
radiometers can have long-term stability of responsivi- 
ty exceeding that of lamps, these filter radiometers can 
act as separate check of the assigned spectral irradi- 
ances. Work is being done to have stable filter radio- 
meters calibrated for spectral irradiance responsivity to 
measure all PEL lamps put through the calibration 
process. 

To perform calibrations in PASCAL 2 of three LUTs, 
against one PWS, requires that a monitoring filter 
radiometer be developed to track the absolute irradi- 
ance of the PWS at a selected wavelength. A silicon- 
detector-based filter radiometer will be used to obtain 
the necessary long-term stability. Ideally the radio- 
meter will operate in the ultraviolet where the lamp 
drift is greatest, to provide the most sensitive tracking 
of PWS stability. The challenge will be to find a detec- 
tor-filter package whose long-term absolute responsiv- 
ity in the ultraviolet is stable to better than 1 %. 

Additional controls to track instrument performance 
include comparing newly assigned spectral irradiances, 
with previous measurements on a customer-submitted 
recalibration, comparing spectral irradiance scales with 
other standards laboratories through international 
comparisons and periodic scale realizations [14]. 

Quality system. A quality system is critical for ensur- 
ing that a measurement is repeatable with the same 
uncertainty budget, independent of operator or time. A 
quality system, such as the ISO 17025 standard imple- 
mented at NIST, provides clear documentation of the 
measurement process to be followed by the calibration 
scientist. It also provides the foundation for implement- 
ing Deming's points 3 and 5 in Table 1. The described 
measurement procedure should be identical to that 
validated through measurement intercomparisons. 

Peer review. Periodic objective technical peer review 
aids the calibration scientist in developing a technical 
rigorous and robust measurement approach and associ- 
ated uncertainty analysis. It aids the elimination of 
aspects of the calibration not well founded in funda- 
mental physics. For a mature calibration service, 
exploring, in detail, the origin of each component of the 
uncertainty analysis typically leads to the conclusion 
that measurement repeats are unnecessary. 

Furthermore, the peer review often reveals steps in 
the measurement process that are done, more, by custom 
rather than for their contribution to the final result. 
Finally an objective analysis can lead to useful sugges- 



tions to improve the overall calibration service. Such 
extensive peer review responds to the spirit of 
Deming's points 1, 5, and 8. 

6. Conclusions 

"Once is Enough" is being implemented throughout 
the calibration programs within the Optical Technology 
Division. Calibration programs involved include 
optical properties of materials measurement, photo- 
metry, and radiometry. The detailed management and 
technical peer review undertaken as part of this effort 
has already led to significant improvements in the 
quality and reliability of the measurements. Such 
improvements include elimination of superfluous steps 
in the process, optimization of controls to ensure instru- 
ment performance, enhanced automation to maximize 
repeatability, and increased understanding of the uncer- 
tainty budget. We anticipate that "Once is Enough" will 
be fully implemented, division-wide, by the end of 
2007. In addition to saving time, as much as 50 %, the 
increased focus on controlling and understanding the 
causes of variability in each step of the process will 
result in an improvement in the overall quality of our 
measurement services, for the benefit of NIST's 
customers. 

We anticipate that the lessons learned within the 
Division on the implementation of "Once is Enough" 
can be disseminated to other calibration services with- 
in NIST and to various calibration laboratories 
involved in supporting the U.S. National Measurement 
System. The concomitant reduction in time and costs 
will allow industry to reduce costs and improve meas- 
urement accuracy, to obtain an increased competitive 
advantage in areas such as manufacturing. Continued 
examination of best practices in manufacturing will, 
likewise, lead to new strategies to improve the per- 
formance and dissemination of NIST calibration 
services. 
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Appendix 1. 

NIST, and formerly NBS, has been performing 
radiometric and photometric calibrations since the 
founding of the Bureau of Standards in the early 20th 
century [15]. The early radiometric and photometric 
scales were based upon flame sources such as candles 
and gas lamps, later progressing to lamps and black- 
bodies. Measurements typically relied upon a visual 
comparison of a test source with a reference source 
[16]. The quantification and standardization of the rela- 
tionship between human visual response and photomet- 
ric and radiometric quantities [17, 18] allowed meas- 
urements of photometric quantities with detectors 
similar to those used in radiometry, but with appropri- 
ate filters to mimic the human visual response. In 1975, 
Blevin and Steiner proposed a redefinition of the 
candela from being based upon blackbody radiation 
to being based upon the amount of optical power at 
555 nm, the peak wavelength for the human visual 
response [19]. The 1979 Conference Generale de Poids 
et Mesures (CGPM) accepted the 1977 Comite 
International de Poids et Mesures (CIPM) recommen- 
dation to adopt this definition for the candela. This 
redefinition initiated a revolutionary change in the 
measurement of photometric quantities. Subjective 
measurements based upon visual inspection were 
replaced by objective measurements performed by 
electro-optical sensors. NIST eliminated such visual 
comparisons in their photometric and radiometric 
measurements and implemented additional step to 
improve radiometric and photometric calibrations [20], 
including the modernization and automation of the 
laboratory facilities and the establishment of an ISO 
Guide 25 compliant quality system [21, 22]. The quali- 
ty system has since been updated to comply with the 
new NIST-wide standards based upon the newer ISO 
Guide 17025 [23]. 
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